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ABSTRACT 

The objectives of this study were to develop 
comparative research data on two library instruction methods in order 
to improve student skills in collecting information, evaluating 
material critically, and communicating effectively through writing, 
as well as to evaluate a methodology for the assessment of library 
instruction. The traditional orientation lecture was provided near 
the beginning of the semester to two sections of a technical writing 
class for undergraduate engineering students. In two other sections, 
instruction in CD-ROM databases, end-user online searching, and 
traditional sources was integrated into the curriculum. A total of 56 
technical writing students completed the requirements of the study 
and provided research papers for data analysis. The Kohl/Wilson 
scoring method was used to check each bibliography to determine: (1) 
if the source use was appropriate for the topic; (2) if the source 
selected represented the need for current versus retrospective 
materials; and (3) the quality of the source for the topic. Results 
indicated that the addition of automated bibliographic sources had 
not made a difference in the bibliographies. The project will be 
continued for four more semesters. (22 references) (MES) 
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Part I — General 
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Part II — Narrative Account 



1. Goals established for the first round of the study were met. A comparison 
was made of the output in the form of bibliographies attached to term 
papers prepared by technical writing students in experimental and control 
groups. The comparison is detailed in the attached narrative. 

2. Changes, approved by telephone contract with Linda Loeb, were made in the 
budget. Equipment changes were necessitated by changes in I Hi's product 
line. Personnel changes were necessitated by schedules, workloads, and 
job changes of those involved. A revised budget was sent to Linda Loeb. 
Changes needed after that submission were approved by telephone and are 
indicated by the figures included on the Financial Status Report. 

3. There is no immediate effect of this Federal project on the population 
served. Rather the first round ha produced information of use to the 
researchers which will be utilized in the second round, now in progress. 
A total of five rounds are included in the project. 

4. This Federal project was not designed to increase resource sharing 
activity, but rather to develop data about two modes of library 
instruction, one of them including the use of automated bibliographic 
information resources. 

5. The assessment of the project is attached. 

6. The project was approved for three years. The second year budget has v aen 
approved and is being implemented. Application for the third year budget 
is in preparation and will be submitted by April 20, 1990. The project, 
currently housed in a temporary laboratory, will be moved to the new 
Science and Engineering Library in the summer of 1990 in time for round 
three of the project in the fall of 1990. 

Approved as a Research and Demonstration project, the following narrative 
includes: 

1. A description of dissemination activities and the results of the 
project's evaluation plan, including data on the impact on technical 
writing students involved. 

2. The innovation use of end types of technology employed in the 
project. 

3. Plans to continue and to build upon this three year project. 
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NARRATIVE 



The project evaluated two methods of instruction provided undergraduate 
engineering students in a technical writing course — the traditional 
orientation lecture and the traditional lecture supplemented with intensive 
course-integrated training in the use of automated information sources 
including CD-ROM and, to a lesser extent, end-user online searching. 
Evaluation focused on bibliographies produced as part of the requirement of a 
term paper for the course. 

Background 

In its report Reform on Campus (in 1972) , the Carnegie Commission on 
Higher Education cited as a factor in the necessity for change in higher 
education a vast increase in human knowledge. It noted that as the rate of 
new knowledge is added to the tremendous existing body, it becomes imperative 
that students learn how to keep abreast of changes both as students and 
throughout life. 1 

Managing the flow and use of literature has been a source of concern to 
the scientific community since the eighteenth century development of the 
specialized journal. 2 The application of automation to publishing with its 
resulting datafiles suggested to Eugene Garfield in the 1960's that the 
fulfilLnent of Vannevar Bush's dream of a scholar's workstation to provide 
such control (universal bibliographic control) , might become possible. 3 By 
1982 electronic publishing had, in fact, made possible over 1,000 publicly 
available databases; by mid-1984 that number had more than doubled. 4 Today 
data are increasingly generated in machine-readable form only. 5 Indeed, the 
scientific journal itself has been questioned as an economical and effective 



means of aammunication when compared with the speed, cost, and more flexible 
distribution of text and data made possible by electronic publishing.* 7 
Numeric databases, available first on magnetic tape and now on optical disk, 
are beginning to be found in libraries beside bibliographic datafiles and 
traditional print materials. 8 

The Carnegie Commission, noting not only the increase in knowledge, but 
also its rapid evolution, indicated that the teaching of existing knowledge 
had became comparatively less important than imparting skills needed for 
continuing self -education through independent study. 9 Among its basic 
recommendations for change in higher education, the commission included the 
need for increased emphasis on the library as an active participant in the 
instructional process. 10 

Academic libraries have long supported user instruction programs. The 
works of Harvie Branscomb, 11 Billy Wilkinson, 12 Patricia Knapp, 13 and Evan 
Farber 14 provide models f >r the profession. In the years since the commission 
report, substantial attention has been directed to various methods deemed 
appropriate for user education. The orientation tour, the introductory 
lecture on basic reference works, and the separate library use course have all 
been touted in the literature. Several notable programs, such as that at 
Earlham College, have integrated library instruction into the curriculum. 
Though this method appears to be the most respected by librarians, there is no 
research data to support claims made for its effectiveness.' 5 

Kohl and Wilson reviewed the literature of bibliographic instruction 
evaluation and observed it to have focused on the following four areas: 

1. student perceptions of the value of such instruction, 



2. changes in student attitudes resulting from such instruction about 
the library and its staff, 

3. student preference for or the effectiveness of one type of 
instructional presentation ever another, and 

4. student ability to answer questions about possible uses of 
specific library resource tools in a test situation. 18 

Indeed, the literature on library instruction is striking in its 
consensus among students, faculty, and librarians of its efficacy in the 
improvement of perceptions about the library and/or librarians, in the growth 
in knowledge of basic information sources, and in the motivation of students 
when instruction is related to a specific objective. The consensus holds 
despite the lack of comparative data relating library instruction to the 
improvement of works of scholarship. Such research efforts, admittedly 
plagued with methodological problems arising from the complexity of human 
interaction, nonetheless, might provide useful insight, if not a one- tc -one 
cause and effect relationship, among instruction and outcome variables. 

vbhl and Wilson reported findings on the relationship between a method 
of library instruction and the quality of bibliographies of student term 
papers as judged in blind evaluations by a librarian and the course 
instructor. Papers to be compared were randomly selected from a number of 
class sections that had received one type of instruction and from an equal 
number of other sections that had received another type. Each bibliography 
was scored on the Lasis of: 

1. Appropriate type of source for the topic (books, 
encyclopedias, etc. ) 

2. appropriate use of current vs. retrospective sources 
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3. quality of sources for the topic (popular vs. 
scholarly sources) 17 

Ratings of the two evaluators were subjected to statistical analysis and 
found to be significantly correlated. The second, and most pertinent, issue 
was whether library instruction content made a difference in the 
bibliographies produced. T-tests were utilized to determine whether there 
were statistically significant differences between the ratings assigned the 
bibliographies of those students who received one type of instruction and 
those who received another. Data derived indicated that there was a 
statistically significant difference in favor of one type of instruction. 18 

Kohl and Wilson developed an innovative and methodologically sound 
approach to the assessment of library use instruction. Although samples 
studied were small, results suggested that data could be gathered which might 
reliably guide judgments in the difficult area of evaluating library 
instruction. 

Project Objectives 
The objectives of the project were to develop comparative research data 
on two library instruction methods (the traditional lecture and course- 
integrated instruction emphasizing the use of CD-ROM and end-user online 
searching) in order to: 

a. improve student skills in collecting information efficiently 
and effectively 

b. improve student skills in evaluating material critically 

c. improve student skills in communicating effectively through 

writing 
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d. evaluate a methodology for the assessment of library 
ins traction 

The University Library system has provided a formal program of library 
instruction for students enrolled in freshman English for the past fourteen 
years. That program, expanded over the past four years through the 
involvement of graduate student interns from the School of 'dbrary and 
Information Studies, provides basic library use training for over 3,000 
students per year. As well, the Library Instruction Department provides 
additional discipline related programs per year. These range in instructional 
requirements, depending upon the requesting faculty member, from tours to 
intensive instruction in the literature of a given discipline. 

Until recently the College of Engineering did not place much stress on 
the use of scientific literature or writing skills at the undergraduate level. 
In response to the rapidity of technical development and resulting technical 
obsolescence within the profession at large, 19 the engineering curriculum has 
become more theory based with concomitant emphases on use of the literature 
and on communication skills. The engineering faculty, eager to encourage such 
improvements, has embraced the library's objectives for this project with 
their students. 

Together the University Library and the engineering and science 
faculties planned for the new Science and Engineering Library to encourage to 
the fullest student development of information gathering and utilization 
skills. To this end an electronic laboratory was planned for the new library 
in cooperation with the School of Library and Information Studies, 
representatives from IBM, and the University Computing Center. 

8 



The electronic laboratory was designed not only to provide education in 
the use of current technologies, but also to examine state-of-the-art 
developments in information retrieval a. id dissemination. Research in the 
electronic laboratory was projected to be the joint responsibility of the 
University Library system and the School of Library and Information Studies 
(SLIS) . SLIS* first doctoral class was enrolled in the fall of 1988, and it 
was expected that the electronic laboratory would provide substantia] 
opportunity for doctoral research. As well, the University Library system and 
SLIS have long shared a master's degree internship program for library service 
students. The electronic laboratory could provide this program new internship 
opportunities. Finally, it was hoped that the information industry might view 
the electronic laboratory as a test site for new products. 

The instructional project utilized the test and evaluation methods 
described by Kohl and Wilson with modifications, in a series of studies to be 
conducted over five semesters. This report analyses round one of that series. 
Because of the difficulties inherent in research efforts to draw conclusions 
concerning human behavior, it was believed that the research results would be 
strengthened considerably by consistent replication. Moreover, it was 
believed that if an instructional method, such as course-integrated library 
instruction utilizing automated information technologies, makes a difference 
in scholarly output of engineering students, such results should be observed 
consistently in varying circumstances over time. The evaluation of 
bibliographies of student papers from various course sections was intended to 
minimize the influence of instructor idiosyncracies. 

For each of five semesters two instruction methods will be utilized with 
students enrolled in technical writing course sections. Technical writing is 

9 
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required for some undergraduate engineering majors. Other engineering majors 
strongly recommend it as an elective. As well, students enrolled in other 
disciplines may take the course as an elective. Typically students enrolled 
in the course are engineering majors with several students in each section 
from other disciplines. In round one, fall semester, 1989, the traditional 
orientation lecture was provided by librarians to two sections near the 
beginning of the semester. These sections received no additional formal 
library instruction. CD-ROM databases and end-user online searching as well 
as instruction in the use of traditional sources was integrated into the 
curriculum of the other two sections and made available throughout the 
semester. 

CD-ROM databases and end-user online databases are two information 
technologies indicative of those ycung engineers may expect to encounter in 
work situations upon completion of undergraduate studies. The strategies and 
skills required for the successful use of these resources can be expected to 
build toward the independent self-education competencies encouraged by the 
Carnegie Commission for undergraduate education. Research data indicated that 
engineering students were the least likely among students enrolled in the 
science and technical disciplines to know about and to utilize the literature 
of their field. 20 Further data indicated that this behavior could be expected 
to continue throughout the careers of practicing engineers. 21 Moreover, the 
need to develop such skills at the undergraduate level was deemed particularly 
important sinc;e most of the students in this population are not expected to 
obtain further formal education. 22 
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CD-ROM technology has gained widespread acceptance in academic 
libraries. 23 studies of its use in libraries demonstrate its enthusiastic 
acceptance by students. * a Locally those experiences have been verified by 
the heavy use made of InfoTrac and such CD-ROM databases as ERIC, ABI Inform, 
and Psychological Abstracts. No studies located addressed the impact or 
relationship of the use of CD-ROM on scholarship. 

Online searching, available for a number of years in academic libraries, 
has been studied widely. Recently vendors have offered lower priced versions 
of their services directly to end-users and to libraries on an after peak-use 
hours schedule. Such services as BRS/After Dark, Knowledge Index, Wil search, 
and CAS Online are increasingly available in libraries, and use reports 
available in the literature indicate that like CD-ROM, these more affordable 
datafiles are gaining enthusiastic acceptance. 26 * 7 Research has not yet 
appeared which considers the impact of the use of these services on student 
scholarship. Investigations of such relationships were viewed by the 
researchers as important. 

Technical writing students involved in the project were trained in the 
use of CD-ROM and online searching and had regular access to these services. 
They met with a librarian for individual guidance on their research topics 
during the course and for guidance and assistance in the use of automated 
resources. 

The purpose of round one was to compare two methods of library 
instruction to determine if one, course-integrated library instruction 
utilizing CD-ROM and end-user online searching, was positively associated with 
improvement in student capabilities in retrieving, evaluating, and 
communicating in writing information on a technical topic. Evaluation of the 
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iupact of automated databases, locally owned and through commercial networks, 
as an element in the improvement of student output is needed by academic 
libraries currently under r^essure to add them as a service. If the desired 
improvements could be attained, the teclinical writing curriculum would be 
revised to reflect such instruction for all students. 

The final objective was the evaluation of the methodology of assessment 
utilized in the project. The method was used with success once on a small 
scale. This project was planned to utilize it in repeated trials to determine 
if, in an area fraught with measurement complexity, similar results might be 
achieved consistently. 

The project utilized personnel from the University Library System 
including librarian instructors whose primary assignment is the provision of 
information services to engineering and science students and faculty. The 
School of Library and Information Studies provided research expertise through 
the services of two faculty members and a doctoral student. The English 
Department cooperated through its Coordinator of Technical Writing and other 
faculty in the preparation, execution and evaluation of library instruction in 
the curriculum. 

Resources utilized included fifteen student workstations each of which 
consists of a PS/2-60 im microcomputer with a CD-ROM player, modem, 5 1/4" 
and 3 1/2" external disk drives, 8513 color monitor. Two printers were 
available and downloading to disk was optional. An instructor workstation 
provided appropriate software testing and record maintenance. An LCD aided 
group instruction. Software included the following CD-ROM databases: NITS, 
General Science Index, Applied Science and Technology Index, and Science 
Citation Index. Online databases were searched through DIALOG'S Knowledge 
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Index. Students developed search skills and strategies and retrieved 
information for evaluation. 

In addition, all students were able to utilize print information 
resources of the Science and Engineering Libraries as well as other library 
services and resources as needed. 

Results 

A total of 56 technical writing students completed all requirements of 
the study and provided research papers suitable for the data analysis phase of 
the first round of the project. 

Students assigned themselves to the four available class sections based 
on schedule needs and preferences. Except for Section I, which was listed 
under the Technical Writing Coordinator's name, instructors were listed as 
"staff" in the class schedule so that without effort students generally could 
not choose class sections based on assumptions about the instructor. It was 
believed that through this process students could be expected to be randomly 
distributed over the four sections by ability, interest in technical writing, 
and other possible performance affecting attributes. 

Twenty-nine students completed project requirements in two control 
course, sections. They received the standard lecture orientation to library 
research. They were given Engineering Library floor plans and encouraged to 
utilize library staff assistance in their term paper research. Twenty-seven 
students completed all project requirements in two experimental course 
sections and received the same traditional lecture orientation as tlit. control 
groups. In addition the experimental sections attended three special sessions 
on the availability and use of CD-ROM and online bibliographic databases. 
They were given the schedule of the electronic laboratory and encouraged to 
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make an appointment with the lab instructor to discuss term paper topics, 
previously selected under the guidance of section instructors, and to select 
and search databases appropriate for topics chosen. Once initiated, 
experimental sections students could utilize the laboratory whenever it was 
open. 

At this point is should be noted that the electronic laboratory was 
originally set-up in temporary quarters pending the completion of the Science 
and Engineering Library. Not only was it not in the library as planned, but 
it was three floors away in a sc. nat remote section of the classroom 
building which also housed the Engineering Library. 

Staffing the electronic laboratory was made more difficult because of 
its location; nonetheless it was open for use an average of thirty hours per 
week during round one of the project. The hours were set after discussion 
with students in the experimental sections. Daytime, evening, and weekend 
hours were established to provide wide use options. Students were noL forced 
to use the room except for the three initial group sessions to learn to use 
CD-RDM and online bibliographic databases. Individualized instruction was 
then available for those who sought it. All but four students took advantage 
of this instruction. 

Data maintained on the use of the room revealed that students overall 
used the laboratory to develop a basic list of potential sources. Once in 
hand, most students were not likely to return to search for additional 
sources. Students expressed enthusiasm for the efficiency, effectiveness, 
speed, and ease of use of such searching. They recognized it as a useful tool 
to be accessed as needed. By and large they were well acquainted with the use 
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of computers in other areas and had utilized them in a variety of learning and 
work tasks and for recreation. 

A few students were < dte enthusiastic and returned to the classroom to 
search databases for technical writing and other assignments. During the 
following semester several students sought permission to continue to utilize 
laboratory resources for new assignments and were granted permission so long 
as it did not interfere with the needs of technical writing students. 

Papers with appended bibliographic were received from 56 students (29 
conurols, 27 experimental) . Identifying cover sheets were removed by the 
project director, leaving on abstract, the paper itself, and its bibliography. 
Each paper was given a number randomly assigned with all papers mixed together 
in one sequence. The papers were read and the bibliographies scored by two 
laboratory instructors, using the Kohl/Wilson system. The original electronic 
laboratory coordinator left the University before the end of the first round, 
and thus did not participate in the scoring or subsequent data analysis. The 
instructors who scored the bibliographies were both librarians, one of them a 
doctoral student in the School of Library and Information Studies. 

The Kohl/Wilson scoring method was used to check each bibliography to 
determine: 

1. if the source use was appropriate for the topic, 

2. if the source selected appropriately represented 
the need for current versus retrospective materials, 

3. the quality of the source for the topic. 

Each of these criteria was scored on a four-point scale in which 0 - 
completely inappropriate, 1 = marginally appropriate, 2 = acceptable, and 3 = 
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superior. The scores for these three items were added together and divided 
by three to give an overall score for each bibliography. 

It was important to develop two scores for each bibliography to 
determine if independently acting instructors would evaluate the 
bibliographies comparably. If scores varied widely, further statistical 
analysis of the date would be useless. Fortunately, Peairson Correlation 
analysis demonstrated a strong relationship between the papers scored by the 
two instructors with a Pearson R of .776 and R2 of .602. and a significance 
level of .01. 

The next and more important concern was whether the two instructional 
methods produced differences in the scores assigned the bibliographies. T- 
tests were used to determine whether there were statistically significant 
differences in bibliographies produced from technical writing instruction that 
included only the traditional library orientation as opposed to those produced 
from the same instruction supplemented with education in and use of automated 
bibliographical resources. This required the development of a single set of 
scores for the control group and for the experimental group. This was 
achieved by adding the scores for each bibliography assigned by the two 
instructors and dividing by two. 

Table I shows the first set of scores comparing the control and 
experimental groups. Using the pooled variance estimate, no statistically 
significant difference was observed. 
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Table I. 

Pooled Variance Estimate — Two Tailed Test 

T-Value Degrees of Freedom 

Significance 

1-40 54 .168 



This result indicated that the addition of automated bibliographic 
sources had not made a difference in bibliographies appended to term papers 
submitted by students in the control and experimental class sections. It was 
learned, however, in the analysis stage that, in fact, seme, perhaps many, of 
the students in the control sections had utilized a CD-ROM database in the 
Engineering Library. Other such databases, available in each campus library, 
may have been used. The actual degree of such use was unknown, but could have 
been a contributing factor in the outcome. This, as well, is not clear and 
presents the need for additional data development and analysis in future 
rounds. A questionnaire has been developed for use in round two, now 
underway, to gather data from control section students about types of sources 
used. It will be administered at the point when term papers are submitted. 

Three technical writing instructors participated in round one. Two 
taught experimental sections, and the other taught the two control sections. 
Table II shows each of the experimental sections as compared to the control 
group. Teacher A shows a statistically significant difference against the 
control group at the .01 level. Teacher B shows no such difference. Further 
analysis revealed no statistically significant difference b, ien the scores 
of the two experimental sections. 
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Table II 




Pooled Variance Estimate 


Two-railed Test Teacher 


T-Value 


Degrees of Freedom 


Sicmif icance T£»vel 


A 


3 . 14 38 


.01 


B 


1.09 43 





An examination of the scores of Teacher A and follow up discussion 
regarding classroom strategies suggested the possibility of a teacher effect 
on student achievement as revealed in the bibliography ratings. Teacher A 
reported factors summarized by Brophy as influencing positive achievement 
among students* more so than did the other experimental section instructor. 
Research over the last fifteen years indicates that some teachers elicit 
higher achievement from comparable students than do other teachers. Two 
common themes cut across the findings. One is that achievement is influenced 
by the amount of time students are engaged in appropriate academic tasks. The 
second is that students learn more effectively when teachers structure new 
information for them and relate it to what students know, and then monitor 
performance and provide corrective feedback during active recitation, drill, 
practice, and application activities. 30 

While no conclusion can be drawn from round one data regarding the 
possibility that teacher A's instructional strategy itself differed 
substantially from that of teacher B, plans have been made to examine this 
phenomena carefully in future rounds in which teacher A may participate. In 
the next such round teacher A will instruct a control section instead of an 
experimental section, and a trained observer will make random sample 
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observations of classroom activities conducted by each participating 
instructor. Though this broadens the scope of analysis devised for the 
project, it is a recognition that factors effecting achievement are 
mult i faceted and difficult to assess. 

No attempt was made to determine the relationship of term paper grades 
to bibliography ratings. Kohl and Wilson found no correlation among these 
scores. 31 They concluded that term paper grades reflect many factors, 
including student ability to use the library, but also such factors as writing 
talent, proofreading, and following instructions. These observations are in 
keeping with a wide range of studies on this topic which show at best low to 
moderate correlations among student grades and achievement. 32 

Finally, initial plans for round one included the availability of word 
processing for the preparation of term papers by students in the experimental 
sections. Because of problems related to the temporary organization and 
location of the electronic laboratory, as well as concerns about staffing and 
equipment security, the electronic laboratory coordinator recommended that 
word processing not be made available in round one. As well, students 
reported access to word processing capabilities in other settings. A decision 
has been made to include word processing in round three when the electronic 
laboratory will be housed permanently in the new Science and Engineering 
Library scheduled to open in the summer of 1990. It is hoped that the more 
convenient location and the availability of word processing, plus the addition 
of new CD-RDM database will result in greater use of the laboratory by 
students enrolled in experimental sections of the technical writing course. 
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Conclusions 

Hie development of useful data regarding effective strategies for 
eliciting positive student achievement is made difficult by the extreme 
complexity of human interaction and the processes through which people learn. 
The use of repeated trials, five for this project, allows for corrections and 
iirprovement in data gathering and analysis over a three year period. This 
strategy may be particularly useful in the development of evaluation practices 
for librarians as they consider the effect of library user education on 
student scholarly output. 

Typically librarians have relied on use perceptions of instruction and 
user comfort with staff and materials as measures of effectiveness. Such 
reports, often flattering, have not provided data relating library instruction 
to the iirprovement of works of scholarship. Moreover, objective measures have 
not been used to examine the usefulness of library materials, some of them, 
particularly current automated databases, very costly in terms of basic 
purchase, staff time, and maintenance. 

Round one of this project does not clarify the picture. It indicated 
the need for additional data gathering and analysis and suggested useful 
additions to. the study. It does indicate that students and technical writing 
instructors, enthusiastic about the efficiency, speed, and effectiveness of 
automated bibliographic searching, regard it as a useful resource to be 
utilized as needed. It does not positively relate the use of automated 
bibliographic resources to a measure of student scholarship. 

Round one also evaluated the methodology developed by Kohl and Wilson to 
judge student scholarship through ratings of bibliographies appended to 
required term papers. No decision was made to revise this method in future 
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rounds though it was decided that additional data should be gathered on the 
implementation of the agreed upon instructional strategy through random 
observation of classroom instructors. 

Generalizations should not be made from round one of the study. Father 
it should be viewed as a steeping stone on a path from which useful data about 
library instruction strategies will be developed and refined over the course 
of the research project. 

Dissemination Plan 
Plans for dissemination of round one include publications to be 
presented to: 



1. 


ERIC, 


2. 


DIALOG OnDisc News, 


3. 


a library journal, 


4. 


a technical writing journal, and 


5. 


an engineering journal. 


As well, 


an oral presentation is scheduled for the Research Forum at the 



Alabama Library Association Annual Meeting in April, 1990. 
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